The ability to record and playback UI interactions benefits many domains, including macro scripting for users of assistive technologies, training, and testing. A significant impediment to reliable recording and playback of UI interaction has been uniquely identifying each UI component in a reliable and performant manner.
In the test domain, thoroughly testing computer programs is very expensive for development enterprises. As a result, some form of automated testing is typically used to save expense, and indeed, often can more consistently test the program than can a large number of independent testers. One type of testing framework is record and playback, in which a tester interacts with a program, with some or all of the user input (e.g., keystrokes and other input such as mouse events) recorded. By recording the input, the same input can be applied to the user interface elements in other versions of the program in a playback operation, whereby the same test can be reused multiple times to find bugs and other problem areas that resulting following a version change.
A first step in automated testing is finding a user interface (UI element) to test. In some cases, this is as simple as finding the running application by its executable filename, and a ControlId for the element. However, especially with multiple frameworks, finding the element is rarely this simple.
For example, many different types of UI-based applications have been deployed, and while the basic concepts of each are similar, their actual implementations are different. In particular, a mandatory property for one application is often optional for another. Further, in the course of adding value, some of the properties that are supposed to be mandatory may have been missed. As a result, in most programs, no single set of properties can be used to uniquely identify all controls.
By way of example, consider a Win32 control such as a button that has an associated ControlId. This ControlId may have been originally intended to be unique to the program, that is, there should be only one control with this value associated with it inside the executable program. However, with large projects that build upon other components, the desired uniqueness intention quickly becomes unmanageable, and is rarely enforced.